A hypothesis

At the storage collections level, alter table is roughly:

1. Grab a snapshot of the frontiers of the previous version, as known to the storage controller.
2. Evolve the schema of the shard.
3. Obtain new handles for the shard. Notably, these frontiers are not advanced: they’re based on whatever’s currently durable in Persist.
4. Generate new metadata for the collection. The initial since hold of this collection is:
   1. the current critical since of the shard. (Read from Persist.)
   2. the overall frontier of the read capabilities obtained in step 1.
5. Atomically modify the controller state: add the new version and point the old version to it. Among other things, this means that only changes to the new version’s frontiers get sent to the background worker and synced back to Persist.

Observations:

• The frontier we get in 3 will generally be behind what we got in 1: since we write sinces back to Persist asynchronously. Even if we ignore the "maybe" in maybe_downgrade_since, there's an interval between when the frontier changes in our in memory state and when it gets dequeued by the background worker and written to consensus state.
• However, it’s also possible for the frontier we get in 3 to be ahead, since this whole setup is not atomic, and we might have downgraded the critical handle in the meantime.

So: suppose we ALTER TABLE right as we downgrade the since of our table from t=10 to t=20, enqueueing but not applying a downgrade. In that case, we might see t=20 in step 1 but t=10 in step 3. Then, as we go to write the new metadata (with an overall since of t=10) into our storage collections state. At this point, the background worker might finally apply the downgrade to t=20, making t=10 permanently unreadable even though the storage controller still thinks it can issue handles at that frontier for the new version of the table.

Great! So far the only nightly errors are either also present on main, or seem like flakes - since they don't alter any tables, so this code wouldn't run - but I'll try and confirm that before merging.